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What  is  the  Problem? 


Time-sensitive  systems  in  uncertain  environments  have  complex 
behaviors.  How  do  we  validate  correct  timing  in  such  systems? 

•  Exact  probabilistic  verification  is  infeasible  due  to  model  size 

•  Black  box  testing  does  not  yield  bounded  predictions 

•  Need  formal  approach  for  dealing  with  uncertainty 

-Accurate,  bounded,  probabilistic  results 
-  In  reasonable  time  even  for  rare  outcomes 

Use  statistical  model  checking  to  do  a  “smart  sampling  of  the  world” 

•  Simulation  captures  both  random  variables  and  timing  (scheduling) 

•  Importance  sampling  “tilts”  input  distributions  for  efficient  probability 
estimation  of  “rare”  events 

Note:  We  use  “probability  estimation”  based  statistical  model 
checking.  There  is  also  a  “hypothesis  testing”  based  version. 
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Statistical  Model  Checking  (SMC) 


Any  system  M  that 
takes  random  inputs 


Probabilistic 
Temporal  Logic 
Formula  0 


Std.Dev. 

Relative  Error  =  — — - 

Mean 


Statistical  Model 
Checker 


Estimated 
Probability  that 
M\=4>  with  relative 
error  RE 


•  System  properties  described  in  formal  language  (UTSL,  BLTL,  etc.) 

•  Property  is  tested  on  “sample  trajectories”  (sequence  of  states) 

•  Each  outcome  can  be  treated  as  a  Bernoulli  random  variable  (i.e.,  coin  flip) 
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Probability  Estimation  with  SMC 


0  0.2  0.4  0.6  0.8  I 


SMC  Basics 

•  Indicator  function  /(x)  =  1  iff  property  holds  for  input  x. 

•  Relative  Error  RE(jp )  =  is  measure  of  accuracy. 

•  Draw  random  samples  from  input  distribution  /(x)  until  target 
Relative  Error  is  met. 

•  Estimated  probability  that  property  holds  is: 

N 


Importance  Sampling 

•  Modify  input  distribution  to  make  rare  properties  more  visible. 

•  Weighting  function  W (x)  maps  solution  back  to  original 
problem. 

•  Reduced  relative  error  with  same  number  of  samples. 

N 
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Osmosis  SMC  Tool 

i 

Osmosis  is  a  tool  for  Statistical 
Model  Checking  (SMC)  with 
Semantic  Importance  Sampling. 

•  Input  model  is  written  in  subset  of  C. 

•  ASSERT()  statements  in  model 
indicate  conditions  that  must  hold. 

•  Input  probability  distributions  defined 
by  the  user. 

•  Osmosis  returns  the  probability  that 
at  least  one  of  the  ASSERT() 
statements  does  not  hold. 

•  Uses  dReal1  solver  to  build  /*(*)■ 

•  Simulation  halt  condition  based  on: 

-  Target  relative  error,  or 

-  Set  number  of  simulations 

1  http://dreal.cs.cmu.edu/ 
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Osmosis  Main  Algorithm 


Generate  approximation  of  fault  region 


x 


Input  space 

7(x):  Indicator  Function 
defines  fault  region 

7*(x):  Abstract  Indicator 
Function  defines  over¬ 
approximation 


.  Conduct  SMC  and  calc,  failure  prob. 


x 


K> 


S< — #  in  Fault 
a)  Praw  =  Tota|  # 

*  6  Fraction  of 

k)  ^  =  64  input  7*0) 
covers 

c)  Failure  prob.  estimate 
is  product  of  two  values 

V  =  PrawP*  =  0-23 
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l*(x )  Generation  Algorithm 


Fault 

Region 


Algorithm: 

1.  Set  the  current  “cube” 
as  the  full  range  of  all 
inputs. 

2.  Apply  dReal  to  the 
current  cube. 

3.  If  the  result  is  “SAT”, 
split  cube  into  two  equal 
probability  cubes  on 
one  variable,  and 
recursively  apply  at 
Step  2. 
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Example:  Air  Hockey  Problem 


Air  Hockey  Problem 

•  Table  with  a  moving  puck  and  a 
fixed  target. 

•  Puck  rebounds  without  friction. 

Inputs 

•  Angle  -  Initial  angle  at  which  puck 
is  hit. 

•  Distance  -  Total  distance  of  travel 
for  puck. 

Failure  Condition 

•  Puck  stops  on  target  (red  dot). 

Challenges 


•  FAIL 


distance 


OK 


•  Multiple  failure  areas  in  input  space. 

•  Complex  structure  of  failure  area. 
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Fault  Map  for  Air  Hockey  Problem 


Fault  map  shows  area  of  input 
space  where  faults  are  located. 

•  Plotted  in  CDF  space. 

•  Green  area  indicates  input  space 
included  in  /*(x). 

•  Red  area  indicates  input  space 
include  in  /(x). 
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Sample  Size  vs  Recursion  Depth  (Air  Hockey) 


Simulation  effort  with  SIS  decreases  exponentially  with  recursion  depth. 
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p*  vs  Recursion  Depth  (Air  Hockey) 


Upper-bound  p*  becomes  more  accurate  as  recursion  depth  increases. 
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Number  of  dReal  Calls 


Effect  of  SIS  Optimizations 


Optimization  1 
No  call  to  dReal  if  first 
child  call  is  unsat. 


Optimization  2 
Use  counter-example 
from  parent  to  avoid 
dReal  calls  on  children. 


Optimization  2  results  in  greatest  benefit  with  factor  of  two  reduction  in  number  of 
calls  to  dReal.  Small  additional  benefit  by  combining  both  methods. 
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Conclusion 


Semantic  Importance  Sampling 

•  Create  approximation  of  fault  region  using  abstraction. 

•  Create  an  alternate  input  distribution  for  importance  sampling. 

•  Level  of  approximation  (recursion  depth)  is  user  tunable. 

•  Can  reduce  SMC  sample  size  by  orders  of  magnitude. 

Osmosis  tool 

•  Applies  semantic  importance  sampling  on  a  C-like  specification. 

•  Uses  the  dReal  SMT  solver  to  build  approximate  fault  region  model. 

•  Can  be  applied  when  there  are  multiple  fault  regions. 

•  Optimization  techniques  can  nearly  halve  number  of  dReal  tests  required. 
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Email:  jhansen@sei.cmu.edu 
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U.S.  Mail 
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Customer  Relations 
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USA 
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